Blackwell Optimality in Markov Decision Processes with Partial Observation by Dinah Rosenberg,

A Blackwell ε-optimal strategy in a Markov Decision Process is a strategy that is ε-optimal for every discount factor sufficiently close to 1. We prove the existence of Blackwell ε-optimal strategies in finite Markov Decision Processes with partial observation. 1. Introduction. A well-known result by Blackwell [3] states that, in any Markov Decision Process (MDP hereafter) with finitely many states and finitely many actions, there is a pure stationary strategy that is optimal, for every discount factor close enough to one. This strong optimality property is now referred to as Blackwell optimality. In this paper we study finite MDPs with partial observations (p.o. hereafter); that is, finite MDPs in which at the end of every stage, the decision maker receives a signal that depends randomly on the current state and on the action that has been chosen, but he observes neither the state nor his daily payoff (see, e.g., [2] and the references in [7]). MDPs with p.o. arise naturally in many contexts, such as models of machine replacement and quality control problems (see [12] and the references therein for this and additional applications), telecommunication networks (see [1] and the references therein), and intra-seasonal decisions of fishing vessel operators [9]. Here we address the problem of existence of Blackwell optimal strategies for a finite MDP with p.o. We prove that, in any such MDP and for every ε, there is a strategy that is Blackwell ε-optimal; that is, ε-optimal for every discount factor close enough to one. The strategy we construct is moreover ε-optimal in the n-stage MDP, for every n large enough. We also provide an example where there is no Blackwell zero-optimal strategy. The standard approach to an MDP with p.o. is to convert it into an auxiliary MDP with full observation and Borel state space. The conditional distribution over the state space given the available information (sequence of past signals and past actions) plays the role of the state variable in the auxiliary MDP. This approach has been developed for instance in [14], [15] and [17]. An alternative state variable is defined in [5]. One then looks for optimal stationary strategies (strategies such that the action chosen in any given stage is only a function of the belief held on